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IMAGE SENSOR AND IMAGE CAPTURE SYSTEM WITH EXTENDED 

DYNAMIC RANGE 



FIELD OF THE INVENTION 

5 The present invention pertains to semiconductor-based image 

sensors with increased dynamic range. 



BACKGROUND OF THE INVENTION 

Solid state image sensors are now used extensively in many types 
10 of image capture applications. The two primary image sensor technologies 

utilized are Charge Coupled Devices CCD and CMOS x-y addressable devices. 
Currently, there exists many different specific embodiments of both technologies, 
including Active Pixel Sensors (APS) and Passive Pixel Sensors (PPS) for CMOS 
x-y addressable devices. All are basically comprised of a set or array of 
1 5 photodetectors that convert incident light into an electrical signal that can be 
readout and used to construct an image correlated to the incident light pattern. 
The exposure or integration time for the array of photodetectors can be controlled 
by well known mechanisms. The signal represents the amount of light incident 
upon a pixel photosite. The dynamic range (DR) of an imaging sensing device is 
20 defined as the ratio of the effective maximum detectable signal level, typically 

referred to as the saturation signal, (Vsat), with respect to the rms. noise level of the 
sensor, (a no ise). This is shown in Equation 1 . 



Equation 1 : Dynamic Range = V sat / Onoise 

25 

Image sensor devices such as charge coupled devices (CCD) that 
integrate charge created by incident photons have dynamic range limited by the 
amount of charge that can be collected and held in a given photosite, (V sat ). For 
example, for any given CCD, the amount of charge that can be collected and 
30 detected in a pixel is proportional to the pixel area. Thus for a commercial device 
used in a megapixel digital still camera (DSC), the number of electrons 
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representing Vsat is on the order of 1 3,000 to 20,000 electrons. If the incident 
light is very bright and creates more electrons that can be held in the pixel or 
photodetector, these excess electrons are extracted by the anti-blooming 
mechanism in the pixel and do not contribute to an increased saturation signal. 
5 Hence, the maximum detectable signal level is limited to the amount of charge 
that can be held in the photodetector or pixel. The DR is also limited by the 
sensor noise level, o n0 ise- Due to the limitations on Vsat, much work has been 
done in CCD's to decrease a no j Se to very low levels. Typically, commercial 
megapixel DSC devices have a DR of 1000:1 or less. 

10 The same limitations on DR also exist for APS and PPS devices. 

The Vsat is limited by the amount of charge that can be held and isolated in the 
photodetector. Excess charge is lost. This can become even more problematic 
with APS and PPS compared to CCD due to the active and passive components 
within the pixel, limiting the area available for the photodetector, and due to the 

1 5 low voltage supply and clocks used in CMOS devices. In addition, since APS 

devices have been used to provide image sensor systems on a chip, the digital and 
analog circuits used on APS devices such as timing and control and analog to 
digital conversion, that are not present on CCD's, provide a much higher noise 
floor on APS devices compared to CCD. This is due to higher temporal noise as 

20 well as possibly quantization noise from the on-chip analog to digital converter. 

In commonly assigned U.S. Patent No. 6,069,377, issued May 30, 
2000, entitled IMAGE SENSOR INCORPORATING SATURATION TIME 
MEASUREMENT TO INCREASE DYNAMIC RANGE, by Prentice et al., 
Prentice discloses the prior art approaches to extending dynamic range of APS 

25 devices, and discloses a new invention to extend dynamic range. This method has 
the disadvantage of requiring more than four transistors per pixel and limits the 
size of the pixel that can be made. In U.S. Patent No. 6,307,195, issued October 
23, 2001, entitled VARIABLE COLLECTION OF BLOOMING CHARGE TO 
EXTEND DYNAMIC RANGE, and U.S. Patent No. 6,486,504, issued November 

30 26, 2002, entitled CMOS IMAGE SENSOR WITH EXTENDED DYNAMIC 
RANGE, both by Guidash, Guidash discloses extending dynamic range by 
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collection of the charge that blooms from the photodetector, and by co-integration 
of the photodetector and floating diffusion within a single pixel. These 
approaches have the potential disadvantage of spatial variation of the 
photodetector saturation level contributing to fixed pattern noise in the sensor, and 
5 does not increase the sensitivity of the sensor. 

Prior art APS devices also suffer from poor sensitivity to light due 
to the limited fill factor induced by integration of active components in the pixel, 
and by loss of transmission of incident light through the color filter layer placed 
above the pixel. 

1 0 From the foregoing discussion it should be apparent that there 

remains a need within the prior art for a device that retains extended dynamic 
range while retaining low fixed pattern noise, small pixel, and high sensitivity. 

SUMMARY OF THE INVENTION 

1 5 The present invention provides a means to control the integration 

separately for any given spatial pattern on the image sensor, and more specifically 
for a pattern that is compatible with one or two dimensions of the kernel in the 
CFA pattern. This is done by providing separate TG or RG busses for pixels in a 
given row or set of rows, or by providing any means to control integration time 

20 separately for a given pattern of pixels in the image sensor array. By doing so, 
valid data is always available for the dark and bright regions of an image 
simultaneously. 

Advantageous Effect Of The Invention 

25 These and other aspects, objects, features and advantages of the 

present invention will be more clearly understood and appreciated from a review 
of the following detailed description of the preferred embodiments and appended 
claims, and by reference to the accompanying drawings. 

30 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 a is a prior art pixel array; 



Fig. lb is another prior art pixel array; 

Fig. 2a is a pixel array of the present invention; 

Fig. 2b is an alternative embodiment of the present invention; 

Fig. 3 is graph graphically illustrating the implementation of Figs. 

2a and 2b; 

Fig. 4a is an illustration of two integration control lines per row; 
Fig. 4b is an illustration of one integration time signal line per row; 

and 

Fig. 5 is a camera for implementing the pixel array of Figs. 2a and 
2b into a preferred commercial embodiment. 

DETAILED DESCRIPTION OF THE INVENTION 

Typical prior art image sensor pixel arrays are shown in Figs. 1 a 
and lb. The image sensor in Fig la can be of any technology type such as CCD or 
CMOS APS. The pixel array 10 in Fig. la comprises a set of photodetectors. The 
integration time is constant for each pixel. The drawback of this approach is that 
if the integration time is long, pixels in the bright areas of an image will become 
saturated and the image details in the bright region will be lost. If the integration 
time is chosen to be short, the image quality in dark regions of the image will be 
poor due to low signal and high noise. The image sensor in Fig lb was disclosed 
in U.S. Patent Application Serial No. 08/960,418, filed July 17, 2002, entitled 
ACTIVE PIXEL SENSOR WITH PROGRAMMABLE COLOR BALANCE, by 
Guidash, in which each color of the pixel array 20 associated with the CFA pattern 
has a separate integration time to achieve charge domain white balance. This has 
the same drawbacks as those cited for the image sensor pixel array in Fig 1 a. 

Referring to Fig. 2a, the image sensor pixel array 30 of the present 
invention includes an array that facilitates different programmable integration 
times, but in a different spatial pattern than that shown in Fig lb. For an x-y 
addressable CMOS image sensor this can be accomplished with separate transfer 
gates or reset gates. For a CCD image sensor this can be accomplished by having 
separate transfer gates. The image sensor pixel array 30 in Fig 2a is constructed to 
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have pixels with two different integration times for mated pairs of rows 40a and 
40b that are correlated with the color filter array pattern pitch or kernel. Pixels 
with long integration times are referred to as fast pixels. Pixels with short 
integration times are referred to as slow pixels. In the case of the Bayer CFA 
5 pattern, this is a two-row pitch. By having separate integration times in this 

pattern, the effective dynamic range of the image sensor is extended as shown in 
Fig. 3. In region 1, low light level region, both the slow and fast pixels of the 
sensor have not saturated. The fast pixels will have signal levels that are well 
above the noise floor. The slow pixels will have signal levels that are within a 

10 predetermined ratio compared to the sensor noise floor. In region 2, both the slow 
and fast pixels have not saturated, and both have adequate signal-to-noise ratio. In 
region 3, high light level regions, the fast pixels have saturated or clipped and do 
not contain valid signal level information. The slow pixels have not saturated and 
do contain valid signal level information with adequate signal to noise ratio. Since 

1 5 the valid information is correlated with the CFA pattern, the missing information 
from the fast pixels can be determined by interpolation of the slow pixels. With 
the separate integration time architecture shown in Fig 3, a single frame capture is 
taken, and spatially adaptive image processing performed. In region 2, standard 
prior art color image processing methods are employed to render an image. For an 

20 area of pixels in the image capture that fall into region 3, interpolation of the slow 
pixels is used to determined the missing signal information in the fast pixels. This 
results in a loss of true MTF in the extremely bright areas of the image, but leads 
to an effectively higher saturation illumination level, Isat. This effectively extends 
the intra-scene dynamic range of the image sensor. Although true spatial 

25 resolution is degraded in the extreme bright regions, the image content that would 
otherwise be lost in the image capture is preserved. 

The sensor architecture of Fig. 2a is designed to provide an 
integration time pattern with two rows of a first integration time, and the two 
adjacent rows with a second integration time. This can be accomplished with any 

30 type of image sensor by having multiple or separate controls for integration time in 
this pattern. For CMOS and other x-y addressable image sensors this can be 
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accomplished simply by having the image sensor timing arranged with two 
separate sets of integration pointers that are applied to the pairs of alternating rows 
signal lines that control integration time. This could be transfer gate lines in each 
row, or reset gates lines in each row, or any other per row signal that is used to 
5 control integration time for that row. In the case of CCD image sensors, this 
requires that the transfer gate interconnects are constructed so that there are 
separate and isolated connections to the transfer gate lines for at least alternating 
pairs of rows. 

A second embodiment of the present invention is shown in the 

10 array in Fig. 2b. In this embodiment, the sensor array 50 is constructed to have 
two separate and programmable integration times in a 2 by 2 pixel pattern 60a and 
60b. In the case of an x-y addressable image sensor technology, this is achieved 
by having multiple signal lines per row that are used to control integration time, 
such as transfer gate or reset gate. These multiple signal lines per row are 

1 5 connected to alternating pairs of pixels to produce the integration time pattern 
shown in Fig, 2b. 

Referring to Fig. 4a, the routing of the multiple signal lines 70 that 
control integration time is shown. One disadvantage with routing multiple signal 
lines 70 to control integration time for each row is reduction of fill factor or a 

20 larger pixel size in order to fit the extra signal lines into the pixel pitch. This is 

overcome by the signal line routing architecture shown in Figure 4b. In this case a 
single integration time control line 80 is used per row, but it is actually routed to 
pixels in two adjacent rows. The signal line 80 in the adjacent row is routed in a 
similar manner to create the integration time pattern shown in Fig. 2b. With this 

25 approach, although a single row of data is readout from the sensor at one time, the 
pixels contained within the data stream are from physically adjacent rows in the 
array. In order to properly reconstruct the image, the interlaced data must be 
corrected in the camera image memory. This is also a feature of the present 
invention. Since either on-chip or in-camera memory can be set up to write data 

30 into two or more row locations, there is no need to have the sensor read out all 
pixels from a physical row at the same time. 
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As previously discussed, this provides an image sensor and image 
capture system with wide intra-scene dynamic range and wide exposure latitude. 
A single image capture can render a full range of image information with 
optimization of the integration time for low light levels without clipping signal 
5 information in the high light regions of an image. This can greatly simplify the 
exposure control system and algorithms in an imaging system since choice of 
exposure or integration time does not need to be as precise. 

It should also be noted that an image capture system using such a 
sensor can be used to measure or determine the dynamic range of a scene to set the 

1 0 two integration times appropriately. During the metering phase of a camera 
system, two widely separated integration times can be used to determine the 
maximum and minimum light levels in the scene. The two integration times can 
then be adjusted to cover the range of illumination in the scene. For example, if 
the dynamic range of the scene to be captured is within the inherent dynamic range 

15 of the image sensor, then the two integration times can be set to the same value. If 
the scene contains a dynamic range that is wider than the true dynamic range of 
the sensor, then the two integration times can be set to match or optimally cover 
the dynamic range of the scene. 

Referring to Fig. 5, there is shown a camera 90 for implementing 

20 the image sensor of the present invention is one of many consumer-oriented 
commercial embodiments. 

The invention has been described with reference to a preferred 
embodiment. However, it will be appreciated that variations and modifications 
can be effected by a person of ordinary skill in the art without departing from the 

25 scope of the invention. 
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